In today's connected world, data is shared between parties all the time. Computers download programs and program updates, organizations share data to cooperate when fulfilling business or other objectives such as those that might be involved in a physical manufacturing process or order request, computer control systems download control instructions and upload reports, audio is exchanged during Voice over IP (VoIP) telephone calls, friends exchange pictures and video over social media, etc. Data transfer, also sometimes called file transfer (even though data other than “files” per se might be the subject of those transfers), thus oftentimes is an integral part of a computer systems, it ensures that information is properly exchanged. File transfer can be especially important for Business-to-Business (B2B) software systems. B2B software systems refer to software systems in enterprises or organizations that talk to each other for information exchange, e.g., when fulfilling orders, sending invoice details, providing product information, etc.
During the early days of software development, file transfer protocols like FTP, SFTP, etc., were used to transfer files between different systems. As is known, the File Transfer Protocol (FTP) is a standard network protocol used for the transfer of computer files between two systems or computers in a network, and SFTP (SSH File Transfer Protocol or Secure File Transfer Protocol) is a network protocol that provides file access, file transfer, and file management over any reliable data stream in a secure manner. SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network. By default, SFTP uses the SSH protocol to authenticate and establish a secure connection. SCP or Secure Copy Protocol also is based on the SSH protocol and provides a means of securely transferring computer files between a local host and a remote host, or between two remote hosts, and SCP still is in use today.
Other technologies also evolved, sometimes in parallel with these basic file transfer protocols. For instance, Electronic Document Interchange (EDI) evolved, making possible the interchange of B2B data in a more structured manner. In modern software systems, numerous technologies are available for facilitating the exchange of data. Web Services, for example, enable information to be exchanged in a more controlled manner.
An increasing number of vendors are offering Managed File Transfer (MFT) products. In general, a MFT product is a software product or solution that manages the (typically secure) transfer of data from one computer to another through a network. MFT products offer additional capabilities on top of file transfers such as, for example, automated data transfers based on specific policies, management of file transfer configurations, monitoring and controlling of the file transfers, error handling, etc. One example MFT product is webMethods ActiveTransfer, which is commercially available from the assignee. MFT products are expected to grow from a valuation of $975.8 million in 2017 to $1,637.3 million by 2024, because of concerns related to reliability, security, support of legacy systems, and/or the like.
Although modern and sophisticated technologies are available for B2B data interchange and other information exchange, MFT products are gaining in popularity and traditional file transfer protocols like FTP, SFTP, etc., are still predominately used in B2B systems and applications for file transfers between enterprises. Yet when it comes to transferring data between systems in a network, traditional file transfer protocols have many disadvantages compared to modern technologies. For example, traditional protocols are technical in nature and deal mostly with the transfer of data between systems in a network. They oftentimes are not “business aware,” whereas more modern technologies sometimes are business- or service-oriented. Also, some modern technologies can integrate with or “talk to” each other, whereas traditional file transfer systems generally cannot operate, or co-operate, in such manners. Many file transfers involve large file transfers including, for example, transfers in the range of a few gigabytes (GB) of data. Large file transfers using traditional transfer protocols are still a challenge to many enterprises and systems because of issues such as, for example, with long transfer times, low bandwidth, data corruption, handling transient failures and resuming the transfer on failure, etc.
Each traditional protocol uses its own different method or technology for authentication, handshake, file transfer, etc., e.g., as documented as part of the corresponding specification. The commands that are exchanged for each protocol generally are different. Also, there are variations for each protocol, as well as newer versions that provide newer capabilities. FIG. 1 is an example signal diagram showing a typical data (file) transfer occurring using a traditional file transfer protocol. As shown in FIG. 1, a client request a connection with a file transfer server, and a connection is established. The client sends authentication data, and the file transfer server indicates whether the authentication is successful. If it is, then the client sends commands to the file transfer server. For instance, the client may request a list of the files, directories, and/or other objects on the file transfer server. The file transfer server responds to those commands. A client might request a file download, triggering the file transfer server to then transfer the file data. Ultimately, the client may request that the connection is closed, and that connection may be closed accordingly.
Each protocol might have additional steps or a custom way to accomplish at least some of the above aspects. For example, FTP will require one socket (or connection) for exchanging commands, and another for transferring data. Also, as alluded to above, the authentication mechanisms, security support, channel for communication, etc., typically will be different for each protocol. It also will be appreciated that the data exchange here is protocol specific, and that this is independent of what data is being exchanged. Irrespective of whether the information exchanged is that of sales information or trading information, big files or small files, partial information or complete information, etc., ultimately, data is converted to files, and the client requests (or downloads) files or uploads files.
More modern day communication architectures are more data-oriented, business-aware, and/or service-oriented technologies. They oftentimes are application programming interface (API) based. REST (Representational State Transfer), for example, is an architectural style that defines a set of constraints to be used for creating web services. Web Services that conform to the REST architectural style, or RESTful web services, provide interoperability between computer systems on the Internet. REST-compliant web services allow the requesting systems to access and manipulate textual representations of web resources by using a uniform and predefined set of stateless operations. In general, a client may request the exact information that it requires, and the server understands the request (what information client is looking for) and sends back the required information. Because the request for data and the response typically happens as and when required, the amount of data that is part of a single data transfer is typically small.
FIG. 2 is an example of a data exchange using a more modern technology such as REST. As shown in FIG. 2, the client requests the data associated with specific sale information, and the server returns the data for the requested information. Here, the request is formulated according to the business need (or data requirement), and the response has only the information requested by the client. Thus, it will be appreciated that this more modern data exchange is more data or business dependent, as opposed to be more protocol dependent.
As noted above, although these more modern technologies and architectures for data exchange oftentimes are easy to use and superior in some of the respects mentioned above, traditional file transfer protocols are still widely used. There are a variety of reasons for this. For example, traditional file transfer protocols are time-tested and proven solutions that in many instances have been running for organizations for many years. They are seen as being reliable and resilient, and organizations oftentimes are resistant to risk this for a newer technology. As another example, there are many legacy systems, mainframes, etc., that are built or otherwise designed to handle data exchanges via more traditional file transfer technologies. As long as these legacy systems are part of the customer IT solution, support for traditional file transfer protocols generally is required for facilitating data exchanges, integration between different systems, etc. As still another example, there are still many use cases or parts of a solution that works on files and thus are not always well suited for modern systems. For instance, scanned images as PDF files, invoices that need to be sent to customers, etc., generally are file-based and file transfer protocols typically will be used accordingly. More modern MFT products (such as, for example, the webMethods ActiveTransfer) provide a lot of management and monitoring capabilities for traditional file transfer protocols, and these technologies sometimes actually enable enterprises to manage, control, and schedule file transfer operations, making further evolution less likely.
As mentioned above, traditional file transfer protocols work on the data or transport level and are not business-aware. Thus, the client generally requests a file rather than business-relevant information, and the server has to make available a set of files that the client can request. A new type of data request from a client will require the server to add the logic to create the new set of files. This is not an easy process, insofar as it oftentimes requires a job running at required intervals to create the required files, and then clean up the files when used, etc. By contrast, in more modern data exchange technologies, it generally is much easier to add a new API or API function, as the API implementation just needs to fetch the data from the backend without necessarily being concerned about file maintenance and other lifecycle activities. Traditional file transfer protocols thus are not as agile as more modern technologies.
In traditional file transfer protocols, even if the client requires only a small part of the data in a file, the client oftentimes will end up downloading the entire file. This behavior generally follows because of the protocols work on the file and the transport level, rather than on the content. This can be especially problematic for large files. For example, files that contains daily sales information in a store, trading information related to a stock exchange, data drawn from a host of Internet-of-Things (IoT) sensors, and/or other Big Data sources generate large amounts of data, but only a comparatively small part of that data may need to be accessed at a given time by a given client. A trading client, for example, might actually want only the trading information for a stock at a particular time, but might be forced to download the entire trading information for the corresponding day because of the way that the protocols are designed and implemented. In more modern API-based technologies, clients can pass the required parameters (e.g., trading time windows, stock price, etc.) so that the server simply sends back to the client only the data that is required and relevant.
More modern day information exchange technologies are designed so that they can interoperate and/or integrate with one another. More current technologies tend to be API-based, and the implementation can be further subdivided into smaller services. For example, the microservices architecture, in which an application is built as a collection of loosely-coupled services that implement business capabilities, is used widely these days. Advantageously, microservices architecture enables service, for instance, to use a different technology to get the required data. Most of the traditional file transfer approaches were created at a time when there is no concept or requirement for microservices architecture. Traditional file transfer protocols are designed as monolithic systems, and such products do not integrate and/or interoperate with other information exchange systems or APIs directly.
Many modern information exchange technologies support API management, which can be used to create and publish the APIs, enforce their usage policies, control access, etc. But because of the way traditional file transfer protocols are defined, and because such products focus on transport technology rather than the data itself, API management cannot be directly applied to these products. There are few MFT products that support API management, but the support is limited to user management, folder or path access, etc., rather than actual APIs.
Because a client using a traditional file transfer protocol cannot request specific information from a large data file, there are many cases where the client ends up receiving large files, sometimes in the size of few GBs. Large file transfer can be a big challenge for any file transfer protocol. If the client and server are located at different geographic locations, if there is network latency between the client and server, etc., then the file transfer can require too much time to complete. Transient failures, network issues, other failures and resume operations, etc., can pose further challenges. It thus becomes more difficult to ensure or implement data integrity checks, etc., in the case of large file transfers.
Generally speaking, there is no way to optimize or accelerate the file transfers in traditional file transfer protocols, especially in the case of large files. It might be technically possible to compress the data or transfer the data differently based on type of files (e.g., for an XML file, a video, etc.), but the protocol specifications currently do not allow any such custom optimization.
It will be appreciated that it would be desirable to overcome the above-identified and/or other problems. For example, it will be appreciated that it would be desirable to incorporate features, including data- and business-aware features and support for APIs, of more modern file transfer paradigms and facilitating evolution beyond these technologies. It also will be appreciated that it would be desirable to leverage the ubiquity of and provide continuing support for more traditional file transfer technologies.
One aspect of certain example embodiments relates to overcoming the above-described and/or other issues. For example, one aspect of certain example embodiments relates to incorporating features, including data- and business-aware features and support for APIs, of more modern file transfer paradigms and facilitating evolution beyond these technologies, while also making it possible leverage the ubiquity of and provide continuing support for more traditional file transfer technologies.
Certain example embodiments relate to a data transfer system including at least first and second computing systems. The first computing system includes first processing resources and a first transceiver, with the first processing resources including at least one first processor and a first memory. The second computing system includes second processing resources and a second transceiver, with the second processing resources including at least one second processor and a second memory. The second computing system is configured to: receive, from the first transceiver of the first computing system via the second transceiver and using a standard communication protocol, a file operation command, the file operation command relating to a data transfer and indicating data to be transferred and/or a preference for how to transfer data; determine whether the first and second computing systems have implemented thereon complementary program logic modules that are configured to facilitate the data transfer in accordance with the file operation command; responsive to a determination that the first and second computing systems do not have implemented thereon complementary program logic modules that are configured to facilitate the data transfer in accordance with the file operation command, cause the data transfer to take place using the standard communication protocol using the first and second transceivers; and responsive to a determination that the first and second computing systems have implemented thereon complementary program logic modules that are configured to facilitate the data transfer in accordance with the file operation command, cause the data transfer to take place using (a) a common communication protocol that is implemented by the complementary program logic modules but different from the standard communication protocol, and/or (b) data selected and/or transformed by the second computing system side program logic module and understandable by first computing system side program logic module.
In certain example embodiments, a data transfer method is provided. The method comprises: receiving, at a server from a client computing system and using a standard communication protocol, a file operation command, the file operation command relating to a data transfer and indicating data to be transferred and/or a preference for how to transfer data; determining whether the server and client computing system have implemented thereon complementary plugins that are configured to facilitate the data transfer in accordance with the file operation command; responsive to a determination that the server and client computing system do not have implemented thereon complementary plugins that are configured to facilitate the data transfer in accordance with the file operation command, causing the data transfer to take place between the server and client computing system and using the standard communication protocol; and responsive to a determination that the server and client computing system have implemented thereon complementary plugins that are configured to facilitate the data transfer in accordance with the file operation command, causing the data transfer to take place using (a) a common communication protocol that is implemented by the complementary plugins but different from the standard communication protocol, and/or (b) data selected and/or transformed by the server-side plugin and understandable by the client computing system side plugin.
In certain example embodiments, a data transfer method is provided. The method comprises: transmitting, to a server from a client computing system and using a standard communication protocol, a file operation command, the file operation command relating to a data transfer and indicating data to be transferred and/or a preference for how to transfer data; responsive to a determination that the server and client computing system do not have implemented thereon complementary plugins that are configured to facilitate the data transfer in accordance with the file operation command, enabling the data transfer to take place between the server and client computing system using the standard communication protocol; and responsive to a determination that the server and client computing system have implemented thereon complementary plugins that are configured to facilitate the data transfer in accordance with the file operation command, enabling the data transfer to take place using (a) a common communication protocol that is implemented by the complementary plugins but different from the standard communication protocol, and/or (b) data selected and/or transformed by the server-side plugin and understandable by the client computing system side plugin.
According to certain example embodiments, the first computing system may be a client and the second computing system may be a server. In this regard, certain example embodiments relate to a client computing system for use as the first computing system and/or a server for use as the second computing system, e.g., for use in relation to the techniques of any of the three previous paragraphs and/or as set forth more fully below.
According to certain example embodiments, the file operation command may be recognizable by the standard communication protocol as a custom command in accordance with the standard communication protocol's specification (e.g., where the standard communication protocol is FTP and the custom command is a SITE command); the file operation command may not be recognizable by the standard communication protocol as a custom command in accordance with the standard communication protocol's specification (e.g., where the standard communication protocol is FTP and the command that is standard for the standard communication protocol is a RETR command); the file operation command may be a command that is standard for the standard communication protocol but is augmented with one or more additional non-standard arguments; etc.—e.g., based on the plugin(s) implemented.
According to certain example embodiments, the preference for how to transfer data that is provided with the file operation command may be tantamount to an inquiry whether the second computing system has one or more second computing system side program logic module implemented thereon.
In addition to the features of the previous paragraphs, counterpart methods, non-transitory computer readable storage media tangibly storing instructions for performing such methods, executable computer programs, and the like, are contemplated herein, as well.
These features, aspects, advantages, and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments of this invention.